Plot marine DA stuff #286

guillaumevernieres · 2023-01-24T19:07:02Z

Description

Addition of a verification script for the marine DA. It does very little for now. I'd like to keep the state space plotting a simple and low level as possible.
No more linking of the DA output in the COM directly. Instead post does the copying of the stuff we want to keep from the cycle and 💣 's DATA.
Don't get to hung-up on the plotting scripts, it will be revisited. It's not great but it does things like plotting slices (horizontal/zonal) of the background error and increment. We'll add EVA stuff later.

fixes godas/453

Testing

JCSDA container
Hera
~~Orion~~ <--- BAD BAD uncooperative MACHINE
We need a dedicated plotting environment for HPC ... And the EVA module (at least on hera) is missing some dependencies requiered by ufsda.

Dependencies

waiting for its companion PR g-w/1270 to be merged before I un-draft this PR.

emcbot · 2023-01-25T19:22:17Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Jan 25 18:20:10 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Jan 25 19:15:11 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Wed Jan 25 19:22:17 UTC 2023
Tests: 85% tests passed, 6 tests failed out of 41
	1309 - test_gdasapp_soca_ana_run (Failed)
	1315 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
	1324 - test_gdasapp_atm_jjob_var_prep (Failed)
	1325 - test_gdasapp_atm_jjob_var_run (Failed)
	1327 - test_gdasapp_atm_jjob_ens_prep (Failed)
	1328 - test_gdasapp_atm_jjob_ens_run (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/286/global-workflow/sorc/gdas.cd/build/log.ctest

emcbot · 2023-01-26T01:22:28Z

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Wed Jan 25 17:47:18 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Jan 25 18:51:48 CST 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Wed Jan 25 19:22:28 CST 2023
Tests: 95% tests passed, 2 tests failed out of 41
	1309 - test_gdasapp_soca_ana_run (Failed)
	1315 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/286/global-workflow/sorc/gdas.cd/build/log.ctest

emcbot · 2023-01-28T04:18:45Z

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Fri Jan 27 21:59:41 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Fri Jan 27 22:18:45 CST 2023
Build: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/286/global-workflow/sorc/log.build

emcbot · 2023-01-28T04:26:31Z

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Fri Jan 27 21:59:41 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Fri Jan 27 22:26:30 CST 2023
Build: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/286/global-workflow/sorc/log.build

guillaumevernieres · 2023-01-28T18:03:15Z

Build issue on Orion:

This is what the `CI` reports

Resetting modules to system default. Reseting $MODULEPATH back to system default. All extra directories will be removed \
from $MODULEPATH.
 .... Building gfs_utils ....
 .... Building WW3 pre and post execs ....
Fatal error in building WW3 pre/post processing.
The log file is in /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/286/global-workflow/sorc/logs/build_ww3_prepost.log
 .... Building forecast model ....
Fatal error in building UFS model.
The log file is in /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/286/global-workflow/sorc/logs/build_ufs.log
 .... Skip building gsi and enkf ....
 .... Building gsi utilities ....
 .... Building GDASApp  ....
 .... Building gsi monitor ....
 .... Building UPP ....
 .... Building ufs_utils ....
 .... Building gldas ....
BUILD ERROR: One or more components failed to build
  Check the associated build log(s) for details.

My own testing

Building GDASApp on orion
Lmod has detected the following error:  Unable to load module because of error when evaluating modulefile:
     /work2/noaa/da/gvernier/pr/global-workflow/sorc/gdas.cd/modulefiles/GDAS/orion.lua: /apps/lmod/lmod/libexec/DirTree.lua:183: interrupted!
     Please check the modulefile and especially if there is a the line number specified in the above message 
While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    GDAS/orion       /work2/noaa/da/gvernier/pr/global-workflow/sorc/gdas.cd/modulefiles/GDAS/orion.lua

Bye bye Orion!!!

I'm marking that PR as ready and ignoring orion.

guillaumevernieres · 2023-01-29T17:36:43Z

It works on orion.
See figures here:

/work2/noaa/da/gvernier/pr/global-workflow/sorc/gdas.cd/build/test/soca/gw/testrun/testjjobs/ROTDIRS/gdas_test/gdas.20180415/12/ocean/vrfy

And here's the chunck of tests relevant to this pr:

(gdasapp) Orion-login-4:/work2/noaa/da/gvernier/pr/global-workflow/sorc/gdas.cd/build/test$ ctest -I 22,27
Test project /work2/noaa/da/gvernier/pr/global-workflow/sorc/gdas.cd/build/test
    Start 22: test_gdasapp_soca_run_clean
1/6 Test #22: test_gdasapp_soca_run_clean ..........................   Passed    0.02 sec
    Start 23: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
2/6 Test #23: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ...   Passed  106.56 sec
    Start 24: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
3/6 Test #24: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ...   Passed   74.18 sec
    Start 25: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
4/6 Test #25: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN ....   Passed   42.17 sec
    Start 26: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
5/6 Test #26: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ...   Passed   10.22 sec
    Start 27: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
6/6 Test #27: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ...   Passed  138.38 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 371.54 sec

... resubmitting the automated ci.

guillaumevernieres · 2023-01-30T15:00:00Z

... last try on orion ...

emcbot · 2023-01-30T17:44:46Z

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Mon Jan 30 09:02:31 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Jan 30 11:27:30 CST 2023
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Mon Jan 30 11:44:45 CST 2023
Tests: 100% tests passed, 0 tests failed out of 42

RussTreadon-NOAA

Hera and Orion GW-RT tests pass. Approve.

ShastriPaturi

Tests pass on hera.
Orion super slow... giving up.

guillaumevernieres · 2023-01-30T21:16:30Z

Thanks for the reviews!

guillaumevernieres added 4 commits January 24, 2023 13:26

wip

977beee

Merge branch 'develop' into feature/vrfy-soca

841b57c

fixed cn

e307640

a few more plots

724f05f

guillaumevernieres added the hera-GW-RT Queue for automated testing with global-workflow on Hera label Jan 25, 2023

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jan 25, 2023

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jan 25, 2023

guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera labels Jan 25, 2023

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jan 25, 2023

guillaumevernieres added the orion-GW-RT Queue for automated testing with global-workflow on Orion label Jan 25, 2023

emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jan 25, 2023

emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jan 26, 2023

Merge branch 'develop' into feature/vrfy-soca

9ca6957

guillaumevernieres removed orion-GW-RT-Failed Automated testing with global-workflow failed on Orion hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jan 26, 2023

guillaumevernieres added 3 commits January 26, 2023 10:48

Merge branch 'develop' into feature/vrfy-soca

eb26976

fixed modules

fee0a5a

code tidy, jet rocks

8189a1a

guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jan 26, 2023

emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jan 26, 2023

emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jan 28, 2023

emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jan 28, 2023

guillaumevernieres requested review from AndrewEichmann-NOAA, ShastriPaturi, CoryMartin-NOAA and RussTreadon-NOAA January 28, 2023 18:06

guillaumevernieres removed the orion-GW-RT-Failed Automated testing with global-workflow failed on Orion label Jan 28, 2023

guillaumevernieres added the orion-GW-RT Queue for automated testing with global-workflow on Orion label Jan 29, 2023

guillaumevernieres marked this pull request as ready for review January 29, 2023 17:37

emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jan 29, 2023

guillaumevernieres added orion-GW-RT Queue for automated testing with global-workflow on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jan 30, 2023

emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jan 30, 2023

emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jan 30, 2023

RussTreadon-NOAA approved these changes Jan 30, 2023

View reviewed changes

ShastriPaturi approved these changes Jan 30, 2023

View reviewed changes

guillaumevernieres merged commit 40dddf9 into develop Jan 30, 2023

guillaumevernieres deleted the feature/vrfy-soca branch February 3, 2023 13:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plot marine DA stuff #286

Plot marine DA stuff #286

guillaumevernieres commented Jan 24, 2023 •

edited

Loading

emcbot commented Jan 25, 2023

emcbot commented Jan 26, 2023

emcbot commented Jan 28, 2023

emcbot commented Jan 28, 2023

guillaumevernieres commented Jan 28, 2023

guillaumevernieres commented Jan 29, 2023

guillaumevernieres commented Jan 30, 2023

emcbot commented Jan 30, 2023

RussTreadon-NOAA left a comment

ShastriPaturi left a comment

guillaumevernieres commented Jan 30, 2023

Plot marine DA stuff #286

Plot marine DA stuff #286

Conversation

guillaumevernieres commented Jan 24, 2023 • edited Loading

Description

Testing

Dependencies

emcbot commented Jan 25, 2023

emcbot commented Jan 26, 2023

emcbot commented Jan 28, 2023

emcbot commented Jan 28, 2023

guillaumevernieres commented Jan 28, 2023

Build issue on Orion:

This is what the CI reports

My own testing

Bye bye Orion!!!

guillaumevernieres commented Jan 29, 2023

guillaumevernieres commented Jan 30, 2023

emcbot commented Jan 30, 2023

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

ShastriPaturi left a comment

Choose a reason for hiding this comment

guillaumevernieres commented Jan 30, 2023

guillaumevernieres commented Jan 24, 2023 •

edited

Loading

This is what the `CI` reports